Systems and methods for displaying augmented anatomical features

ABSTRACT

A method, user device, and system for displaying augmented anatomical features is disclosed. The method includes detecting a target individual having a body. The method also includes tracking the target individual. The method further includes displaying, on a display, a visual representation of the body. The method further includes identifying a plurality of reference markers on the visual representation of the body. The method also includes determining, at a processor, an anatomical profile of the target individual based on the plurality of reference markers. The anatomical profile includes a plurality of anatomical features. The method further includes displaying, on the display, graphical representations of the anatomical features overlaid on the visual representation of the body. The graphical representations of the anatomical features are oriented on the visual representation of the body based on the anatomical profile.

TECHNICAL FIELD

This disclosure relates generally to a method, system and user devicefor displaying augmented anatomical features, and more particularly to asystem and method for displaying augmented anatomical featuresapproximating one or more characteristics of a user's anatomicalfeatures.

BACKGROUND

This section provides background information related to the presentdisclosure and is not necessarily prior art.

Augmented reality technology has the ability to alter, or augment, auser's view of the surrounding environment by overlayingcomputer-generated images onto the user's view of the real world,creating a composite view consisting of both real and virtual elements.Augmented reality offers the user an enriching experience by augmenting,via overlaid digital content, the user's perception of their environmentand their immediate surroundings. The user may augment their viewthrough various electronic devices, such as wearable technology (e.g.,headsets, glasses, smart watches, etc.), tablets, laptops, mobiledevices, or other devices. The user can use these electronic devices toaugment their perception of their environment by overlaying, forinstance, information about their surroundings, or graphical images toenhance their perception of their current environment.

Augmented reality can be used in a variety of environments by a varietyof users to educate each user about their surroundings. For example, arailyard worker can wear augmented reality glasses that allow them toview information about trains in the railyard, or a biologist may useaugmented reality to identify different species of plants surroundingthem.

Healthcare professionals, such as doctors and nurses, are in continuousneed of technological assistance in order to treat their patients.Particularly, healthcare professionals constantly need to obtain andaccumulate data on their patients in order to assess the best treatmentplan for the patient. Healthcare professionals would greatly benefitfrom using augmented reality to gather data on their patients. Whileknown augmented reality technology has been used for healthcareprofessionals to gather patient data, a continuous need for improvementremains in the pertinent art.

SUMMARY

This section provides a general summary of the disclosure, and is not acomprehensive disclosure of its full scope or all of its features.

The instant disclosure provides various methods, user devices, andsystems for displaying augmented anatomical features. One aspect of thedisclosure provides a method. The method includes detecting a targetindividual. The target individual has a body. The method furtherincludes tracking the target individual. The method further includesdisplaying, on a display, a visual representation of the body. Themethod further includes identifying a plurality of reference markers onthe visual representation of the body. The method further includesdetermining, at a processor, an anatomical profile of the targetindividual. The anatomical profile of the target individual isdetermined based on the plurality of reference markers. The anatomicalprofile includes a plurality of anatomical features. The method furtherincludes displaying, on the display, graphical representations of theanatomical features overlaid on the visual representation of the body.The graphical representations of the anatomical features are oriented onthe visual representation of the body based on the anatomical profile.

Implementations of the disclosure may also include one or more of thefollowing features. In some implementations, the plurality of referencemarkers correspond to at least one of a navel, a portion of a sternum, aportion of a hip, a portion of a collarbone, or a portion of a shoulder.

In some implementations, the anatomical profile of the target individualis further based on a plurality of data corresponding to the body.

In some implementations, the plurality of anatomical features includesat least one of organs, bones, muscles, or blood vessels.

In some implementations, the plurality of reference markers areidentified by a user interacting with the display.

In some implementations, the anatomical profile of the target individualincludes a default anatomical profile that is modified based on theplurality of reference markers.

According to another aspect of the disclosure, a user device isprovided. The user device includes a display. The user device furtherincludes data processing hardware in communication with the display. Theuser device further includes memory hardware in communication with thedata processing hardware. The memory hardware stores instructions thatwhen executed on the data processing hardware cause the data processinghardware to perform operations. According to this aspect, the operationscarry out a method. The method includes detecting a target individual.The target individual has a body. The method further includes trackingthe target individual. The method further includes displaying, on thedisplay, a visual representation of the body. The method furtherincludes identifying a plurality of reference markers on the visualrepresentation of the body. The method further includes determining, ata processor, an anatomical profile of the target individual. Theanatomical profile of the target individual is determined based on theplurality of reference markers. The anatomical profile includes aplurality of anatomical features. The method further includesdisplaying, on the display, graphical representations of the anatomicalfeatures overlaid on the visual representation of the body. Thegraphical representations of the anatomical features are oriented on thevisual representation of the body based on the anatomical profile.

This aspect may also include one or more of the following features. Insome implementations, the plurality of reference markers correspond toat least one of a navel, a portion of a sternum, a portion of a hip, aportion of a collarbone, or a portion of a shoulder.

In some implementations, the anatomical profile of the target individualis further based on a plurality of data corresponding to the body.

In some implementations, the plurality of anatomical features includesat least one of organs, bones, muscles, or blood vessels.

In some implementations, the plurality of reference markers areidentified by a user interacting with the display.

In some implementations, the anatomical profile of the target individualincludes a default anatomical profile that is modified based on theplurality of reference markers.

According to another aspect of the disclosure, a system is provided. Thesystem includes a user device. The user device includes a first display.The user device further includes data processing hardware incommunication with the first display. The user device further includesmemory hardware in communication with the data processing hardware. Thememory hardware stores instructions that when executed on the dataprocessing hardware cause the data processing hardware to performoperations. According to this aspect, the operations carry out a method.The method includes, detecting a target individual. The targetindividual has a body. The method further includes tracking the targetindividual. The method further includes displaying, on the firstdisplay, a visual representation of the body. The method furtherincludes identifying a plurality of reference markers on the visualrepresentation of the body. The method further includes determining, ata processor, an anatomical profile of the target individual. Theanatomical profile of the target individual is determined based on theplurality of reference markers. The anatomical profile includes aplurality of anatomical features. The method further includesdisplaying, on the first display, graphical representations of theanatomical features overlaid on the visual representation of the body.The graphical representations of the anatomical features are oriented onthe visual representation of the body based on the anatomical profile.The system further includes a wearable in communication with the userdevice. The wearable includes a second display configured to displaygraphical representations of the anatomical features overlaid on thevisual representation of the body. The graphical representations of theanatomical features are oriented on the visual representation of thebody based on the anatomical profile.

This aspect may also include one or more of the following features. Insome implementations, the plurality of reference markers correspond toat least one of a navel, a portion of a sternum, a portion of a hip, aportion of a collarbone, or a portion of a shoulder.

In some implementations, the anatomical profile of the target individualis further based on a plurality of data corresponding to the body.

In some implementations, the plurality of anatomical features includesat least one of organs, bones, muscles, or blood vessels.

In some implementations, the plurality of reference markers areidentified by a user interacting with the display.

In some implementations, the wearable is further configured to detectand track the target individual independently of the user device.

In some implementations, the anatomical profile of the target individualincludes a default anatomical profile that is modified based on theplurality of reference markers.

The details of one or more implementations of the disclosure are setforth in the accompanying drawings and the description below. Otheraspects, features, and advantages will be apparent from the descriptionand drawings, and from the claims.

DESCRIPTION OF DRAWINGS

The drawings described herein are for illustrative purposes only ofselected configurations and not all possible implementations, and arenot intended to limit the scope of the present disclosure.

FIG. 1A illustrates an exemplary augmented reality system in accordancewith the principles of the present disclosure.

FIG. 1B illustrates an exemplary use of a user device of the augmentedreality system of FIG. 1A.

FIG. 1C illustrates an exemplary use of the augmented reality system ofFIG. 1A in accordance with the principles of the present disclosure.

FIG. 1D illustrates an exemplary use of the augmented reality system ofFIG. 1A in accordance with the principles of the present disclosure.

FIG. 1E illustrates an exemplary use of the augmented reality system ofFIG. 1A in accordance with the principles of the present disclosure.

FIG. 1F illustrates an exemplary use of the augmented reality system ofFIG. 1A in accordance with the principles of the present disclosure.

FIG. 2 is a block diagram illustrating an exemplary augmented realitysystem in accordance with the principles of the present disclosure.

FIG. 3 is a block diagram illustrating an exemplary method fordisplaying augmented anatomical features in accordance with theprinciples of the present disclosure.

FIG. 4 is a schematic view of an example electronic device executinginstructions for displaying augmented anatomical features in accordancewith the principles of the present disclosure.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

Some of the implementations of the disclosure will be described morefully with reference to the accompanying drawings. Exampleconfigurations are provided so that this disclosure will be thorough,and will fully convey the scope of the disclosure to those of ordinaryskill in the art. Specific details are set forth such as examples ofspecific components, devices, and methods, to provide a thoroughunderstanding of configurations of the present disclosure. It will beapparent to those of ordinary skill in the art that specific detailsneed not be employed, that example configurations may be embodied inmany different forms, and that the specific details and the exampleconfigurations should not be construed to limit the scope of thedisclosure.

Example implementations provide methods, user devices, and systems fordisplaying augmented anatomical features. An augmented reality device,such as an augmented reality headset or other electronic device (e.g., aphone, a tablet computing device, or other computer), may be used tooverlay computer-generated or virtual images onto a real world view.Particularly, a healthcare professional, such as a doctor or nurse, mayuse an augmented reality device to view virtual images of anatomicalfeatures of a human body overlaid on a target individual, such as apatient, when the target individual is in view of the healthcareprofessional. The augmented reality device may project the virtualimages onto a display of the augmented reality device such that thevirtual images of anatomical features approximate one or morecharacteristics (e.g., size, location, shape, etc.) of the targetindividual's actual anatomical features. For example, the augmentedreality device may project the virtual images onto a display of theaugmented reality device such that the virtual images are located overan approximated appropriate location of the target individual's actualanatomical features according to the anatomy of the target individual.The virtual images may assist a healthcare professional in moreaccurately assessing a treatment plan or otherwise treating the patientby enhancing the health care professional's visualization of thepatient's body.

Referring now to FIG. 1A, an exemplary augmented reality system 10,including one or more augmented reality device(s) 102, is shown. As willbe described in more detail below, a user 104 may use the augmentedreality device(s) 102 in a healthcare environment 100 to enhance theuser's view of a target individual 106. For example, the user 104 may bea doctor, the target individual 106 may be a patient, and the healthcareenvironment 100 may be a doctor's office, such that the doctor is ableto examine the patient in the doctor's office. In another example, theuser 104 may be a paramedic providing emergency treatment to a targetindividual 106 in a healthcare environment 100 of an ambulance. Whilethe user 104 is generally shown and described herein as being ahealthcare professional (e.g., a doctor, nurse, physical therapist ortrainer, paramedic, medical assistant, pharmacist, etc.), and the targetindividual 106 is generally illustrated and described herein as being ahealthcare patient, the user 104 or target individual 106 may includevarious other persons within the scope of the present disclosure. Forexample, the individual 106 may be an athlete, student, or otherindividual that has a body and is subject to examination or study byanother user 104. In this regard, the augmented reality device(s) 102may be used in a wide range of settings by a variety of users 104 toexamine a target individual 106 in a variety of environments, includingthe healthcare environment 100.

The augmented reality device 102 may include an image capture device 108and a display 110. As will be described in more detail below, duringuse, the image capture device 108 may obtain data about the healthcareenvironment 100 and, particularly, the target individual 106 located inthe healthcare environment 100, and the display 110 may display, foruser 104 to view, a composite view of the healthcare environment 100overlaid with virtual images generated by the augmented reality device102. In some implementations, the system 10 includes a first augmentedreality device 102 a and a second augmented reality device 102 b. Thefirst augmented reality device 102 a may include a smartphone, tabletcomputer, or other suitable mobile computing device, and the secondaugmented reality device 102 b may include an augmented reality headset.References herein to the augmented reality device 102 will be understoodto apply equally to the first augmented reality device 102 a and/or thesecond augmented reality device 102 b.

The first augmented reality device 102 a may include an image capturedevice 108 a (e.g., a camera) and a display 110 a (e.g., a screen).During use, the image capture device 108 a may capture images of thehealthcare environment 100 and, particularly, the target individual 106.The screen 110 a of the first augmented reality device 102 a may be usedto display a composite view of the healthcare environment 100, capturedby the camera 108 a, and overlaid with virtual images generated by thefirst augmented reality device 102 a. The first augmented reality device102 a may include a keyboard, mouse, microphone, camera 108 a, ortouchscreen for allowing user 104 to input data to the first and/orsecond augmented reality device 102 a, 102 b.

The second augmented reality device 102 b may include an image capturedevice 108 b (e.g., a camera) and a display 110 b (e.g., an eyepiecelens). During use, the image capture device 108 b may capture images ofthe healthcare environment and, particularly, the target individual 106.The display 110 b may display a composite view of the healthcareenvironment 100, captured by camera 108 b and/or the camera 108 a, andoverlaid with virtual images generated by the second augmented realitydevice 102 b. The second augmented reality device 102 b may include atrackpad 111, camera 108 b, microphone, eye tracking device, or gesturetracking device for allowing user 104 to input data to the first and/orsecond augmented reality device 102 a, 102 b. For example, the user 104may input data and otherwise interact with the second augmented realitydevice 102 b by touch via trackpad 111; spoken commands via amicrophone; eye gestures via the camera 108 b; positional tracking ofhands or other body parts via the camera 108 b; hand gesture trackingvia the camera 108 b; or positional tracking of objects such as wands,styluses, pointers, or gloves via the camera 108 b.

Though the example shown depicts augmented reality device 102 as a firstaugmented reality device 102 a or second augmented reality device 102 b,it should be noted that augmented reality device 102 may be any device(e.g., augmented reality glasses, augmented reality helmet, tablet,etc.) capable of overlaying computer-generated or virtual images onto areal word view.

With reference now to FIG. 1B, an exemplary augmented reality device isshown. Although FIG. 1B depicts augmented reality device 102 as thefirst augmented reality device 102 a displaying visual data on screen110 a, it should be noted that, similarly, second augmented realitydevice 102 b may also display visual data on eyepiece display 110 b.Augmented reality device 102 may detect the target individual 106 byusing image capture device 108. Augmented reality device 102 may thendisplay, on display 110, a visual representation 105 of a body of thetarget individual 106. In some implementations, the visualrepresentation 105 is a live (e.g., real time) image of the targetindividual 106. In other implementations, the visual representation 105is a still image (e.g., a photograph) of the target individual 106. Insome implementations, the image capture device 108 includes an infraredcamera that uses infrared laser scatter beam technology, for example, tocreate a three-dimensional visual representation 105 of the targetindividual 106.

The augmented reality device 102 may identify (e.g., assign) one or morereference markers 112 on the visual representation 105. As describedabove, in some implementations, the reference markers 112 are identifiedon a three-dimensional visual representation 105 created using infraredlaser scatter beam technology. Each reference marker 112 may correspondto a particular part of, or location on, the body of the targetindividual 106. In some implementations, the augmented reality device102 assigns the reference marker(s) 112 by detecting an input (e.g.,touch, hand gesture, etc.) from the user 104 corresponding to one ormore particular parts of the body of the target individual 106. Inparticular, the reference markers 112 may be identified by the user's104 interaction with the augmented reality device 102. For example, insome implementations the user 104 touches the screen 110 a at locationscorresponding to each reference marker 112. In other implementations,the augmented reality device 102 b receives an input from the user 104via the camera 108 b, or the trackpad 111 corresponding to eachreference marker 112. For example, the camera 108 b may capture thelocation of the user's 104 hand at locations corresponding to eachreference marker 112.

In some implementations, the augmented reality device 102 recognizes andassigns the reference marker(s) 112 to one or more particular parts ofthe body (e.g., facial features) of the target individual 106. Forexample, the image capture device 108 may include an infrared camerathat uses infrared laser scatter beam technology, for example, torecognize and assign the reference marker(s) 112 to the one or moreparticular parts of the body (e.g., facial features) of the targetindividual 106. In particular, the image capture device 108 may be ableto create a three-dimensional reference map of the face of the targetindividual 106 and compare the three-dimensional reference map toreference data stored in a storage resource of the augmented device 102,such as the storage device 430 (FIG. 4). The augmented reality device102 may use the infrared camera of the image capturing device 108 toidentify the reference markers 112 on the face of the target individual106. The augmented reality device 102 may identify the reference markers112 on the lips, corners of the mouth, tip of the nose, or ears of thetarget individual 106. For example, the augmented reality device 102 mayidentify the reference markers 112 based on input (e.g., touch, handgesture, etc.) from the user 104. As will be explained in more detailbelow, in some implementations, the augmented device 102 uses theidentification information from the infrared camera, along with theidentified referenced markers 112 based on the input from the user 104,to transmit data corresponding to the location of the reference markers112 to a processing module (e.g., processor 410 of FIG. 4) to allow theaugmented reality device 102 to advantageously give more individualizedand specific estimates of the location of various anatomical features onthe body (e.g., face) of the target individual 106, including theunderlying blood vessels, nerves, and muscles.

In some implementations, the augmented reality device 102 identifies andassigns the reference marker(s) 112 by using machine learning orartificial intelligence algorithms to identify particular parts of thebody of the target individual 106. The augmented reality device 102 mayassign the locations of the reference markers 112 on the targetindividual 106 based on the locations of similar reference markers 112on one or more other target individuals 106. The augmented realitydevice 102 may use machine learning or artificial intelligencealgorithms to identify the target individual 106 as being a human bodyby detecting a silhouette of the target individual 106, recognizing bodyparts of the detected silhouette (e.g., limbs, crotch, armpits, orneck), and then determining the location of, and assigning, referencemarkers 112 based on the recognized body parts.

In the example shown, a first reference marker 112 a corresponds to anavel of the target individual 106. A second reference marker 112 bcorresponds to a portion of a right shoulder of the target individual106. A third reference marker 112 c corresponds to a portion of a leftshoulder of the target individual 106. A fourth reference marker 112 dcorresponds to a portion of a collarbone of the target individual 106. Afifth reference marker 112 e corresponds to a portion of a left hip ofthe target individual 106. A sixth reference marker 112 f corresponds toa portion of a right hip of the target individual 106. A seventhreference marker 112 g corresponds to a portion of a sternum of thetarget individual 106. Reference markers 112 a-112 g do not represent anexhaustive list of all reference markers, but rather an exemplary listof reference markers that may be identified by augmented reality device102. Furthermore, augmented reality device 102 may identify otherreference markers in addition to reference markers 112 a-112 g, such asreference markers corresponding to a neck, a nose, eyes, a mouth, knees,ankles, a gluteal fold, shoulder blades, wrists, or elbows, withoutdeparting from the teachings herein. Augmented reality device 102 mayalso omit one or more reference markers 112 a-112 g, without departingfrom the teachings herein.

The augmented reality device 102 may determine an anatomical profile ofthe target individual 106. The anatomical profile may include aplurality of characteristics corresponding to the individual 106. Insome implementations, the anatomical profile includes or is based on aplurality of target data, such as age or sex of the target individual106. In some implementations, the augmented reality device 102determines the anatomical profile based on an input (e.g., touch, handgesture, etc.) from the user 104. In other implementations, theaugmented reality device 102 uses machine learning or artificialintelligence algorithms to determine the anatomical profile. Forexample, the augmented reality device 102 may determine the anatomicalprofile based on a plurality of target data (e.g., the plurality ofreference markers 112) received by the augmented reality device 102.

Referring now to FIG. 1C, exemplary augmented reality devices are shown.The augmented reality devices 102 a, 102 b may display a graphicalrepresentation 113 of the target individual 106. In someimplementations, the augmented reality device 102 may display thegraphical representation 113 even when the target individual 106 isfully clothed. This is advantageous as it saves time because the targetindividual 106 does not have to remove their clothing. The graphicalrepresentations 113 of the target individual may include one or morevirtual images 114 a-114 c of anatomical features. For example, thevirtual images 114 a-114 c may be virtual images of internal (e.g.,bones or other organs) or external (e.g., skin or contours thereof)anatomical features. In some implementations, the augmented realitydevice 102 uses three-dimensional data corresponding to the targetindividual 106 to generate the graphical representations 113. In someimplementations, the graphical representations 113 include virtualimages 114 of a detailed underlying anatomy of the face of the targetindividual 106.

A method of displaying the graphical representation 113 may includevarious steps. For example, the method of displaying the graphicalrepresentation 113 may include identifying one or more reference markers112 (FIG. 1B) with the augmented reality device 102 (e.g., 102 a, 102b). The augmented reality device 102 may also determine the distancefrom each reference marker 112 to each of the other reference markers112 and transmit data corresponding to the distances to a processor(e.g., processor 410 of FIG. 4). The augmented reality device 102 mayfurther perform a look up in a database (e.g., database 218 in FIG. 2)with data corresponding to reference markers 112 (e.g., memory 420 ofFIG. 4). The augmented reality device 102 may use the data correspondingto reference markers 112, and in some implementations the plurality oftarget data, to determine data corresponding to the anatomical features'characteristics (e.g., size, location, etc.). For example, the augmentedreality device 102 may use the reference markers 112 and thethree-dimensional visual representation 105 of the target individual 106created from infrared laser scatter beam technology of the image capturedevice 108 to create the graphical representation 113, including thevirtual images 114 a-c of the anatomical features. In particular, theaugmented reality device 102 may transmit the data corresponding to theanatomical features' characteristics to the processor (e.g., processor410 of FIG. 4) and display the graphical representation 113, includingthe virtual images 114 a-c of the anatomical features, on the display110 at a location corresponding to the target individual 106 (see FIG.1C).

As illustrated in FIG. 1C, the second augmented reality device 102 b maydisplay the graphical representation 113 on the eyepiece display 110 b,and the first augmented reality device 102 a may display the graphicalrepresentation 113 on screen 110 a. As previously described, thegraphical representation 113 displayed on screen 110 and overlaid on topof visual representation 105 of the target individual 106 may includecomputer-generated virtual images (e.g., the virtual images 114 a-c ofthe anatomical features). In the example shown, augmented reality device102 displays, on display 110, the graphical representation 113 includesvirtual images 114 a-114 c each representing an organ, bone, orstructure in the human body.

A first virtual image 114 a represents a ribcage of the human body.Virtual image 114 a is overlaid on the visual representation 105 of thetarget individual 106 at a location approximating where the ribcage ofthe target individual 106 is located. A second virtual image 114 brepresents intestines of the human body. Virtual image 114 b is overlaidon the visual representation 105 of the target individual 106 at alocation approximating where the intestines of the target individual 106are located. A third virtual image 114 c represents a pancreas of thehuman body. Virtual image 114 c is overlaid on the visual representation105 of the target individual 106 at a location approximating where thepancreas of the target individual 106 is located. Virtual images 114a-114 c do not represent an exhaustive list of all virtual image, butrather an exemplary list of virtual images that may be displayed by theaugmented reality device 102 in the graphical representation 113.Furthermore, the augmented reality device 102 may display other virtualimages in addition to the virtual images 114 a-114 c, or may omit one ormore virtual images 114 a-114 c, without departing from the teachingsherein.

The graphical representation 113, in combination with the visualrepresentation 105, enhances the view and experience of the user 104 bycreating a composite view of both real and virtual images on the display110. The user 104 may view, through the display 110, the visualrepresentation 105 of the target individual 106 with the virtual images114 a-114 c of organs and other anatomical features represented by thegraphical representation 113 of the target individual 106. Theanatomical features may include organs, bones, muscles, blood vessels,tendons, ligaments, or nerves. In some implementations, the virtualimages 114 a-114 c are not actual images of the internal organs, bones,or other bodily structures of target individual 106, but rather arerepresentative depictions (e.g., illustrations) of those bodilystructures. Allowing the augmented reality device 102 to storerepresentative virtual images 114 a-114 c that can be used for anytarget individual 106 is advantageous because it requires less space(e.g., memory 420) for storing the virtual images 114 a-114 c, allowingthe augmented reality device 102 to operate and display the virtualimages 114 a-114 c at a faster speed.

In some implementations, the augmented reality device 102 may use theimage capture device 108 to take a still picture or record a video ofthe target individual 106. The augmented reality device 102 may thenoverlay virtual images 114 of anatomical features onto the still pictureor recorded video of target individual 106. The augmented reality devicemay then display the still picture or recorded video with the overlaidvirtual images 114 onto the display 110. This may be advantageous,particularly for educational purposes, as it assists a healthcareprofessional in educating a patient about their own anatomy by showingthe patient how their own individual anatomy approximately works. Inother implementations, the augmented reality device 102 may be able totake a picture or record a video of the target individual 106 with thevisual representation 105 that includes the virtual images 114 of organsand other anatomical features. The augmented reality device 102 maystore the picture or recorded video in a storage resource, such as thestorage device 430 (FIG. 4).

With reference now to FIGS. 1D-1F, in some implementations, theaugmented reality device 102 (e.g., 102 a, 102 b) may display theanatomical profile of a target (e.g., target individual 106) based oncertain anticipated changes to the target individual 106. FIG. 1D showsa visual representation 105′ of an alternative target individual 106′displayed on display 110. The augmented reality device 102 may determinethe visual representation 105′ of the alternative target individual 106′with the image capture device 108. As illustrated in FIG. 1E, similar tothe visual representation 105 of FIG. 1B, the augmented reality device102 may identify a plurality of reference markers 112 on the visualrepresentation 105′. In some implementations, the augmented realitydevice 102 may determine an anatomical profile of the alternative targetindividual 106′. The anatomical profile may include a plurality ofcharacteristics corresponding to the alternative target individual 106′.As described above in reference to FIG. 1B, the anatomical profile maybe based on a plurality of target data, input from the user 104, ormachine learning or artificial intelligence algorithms. In someimplementations, the anatomical profile may also be based on certainchanges that the alternative target individual 106′ may undergo. Forexample, the anatomical profile may be based on future weight loss orweight gain that the alternative target individual 106′ will, or desiresto, undergo.

FIGS. 1E and 1F show the augmented reality device 102 displaying thevisual representation 105′ of the alternative target individual 106′ ondisplay 110. The augmented reality device 102 displays the graphicalrepresentation 113 of the alternative target individual 106′, thegraphical representation 113 including virtual images 114 of anatomicalfeatures. In the example shown at FIG. 1F, the augmented reality device102 displays the visual representation 105′ of the alternative targetindividual 106′ if the alternative target individual 106′ had lostweight (FIG. 1F). The augmented reality device also overlays thegraphical representation 113 that includes the virtual images 114 ofanatomical features. The virtual images 114 of anatomical features maybe based on the anatomical features of the alternative target individual106′ if the alternative target individual 106′ lost weight. Furthermore,the visual representation 105′ (FIG. 1F) of the alternative targetindividual 106′ may be based on if the alternative target individual106′ lost weight.

Though FIG. 1F depicts the augmented reality device 102 displaying thevirtual images 114 of anatomical features based on if the alternativetarget individual 106′ lost weight, it should be noted that theaugmented reality device may display virtual images 114 based on othercertain changes, such as gaining weight, becoming pregnant, undergoingreconstructive or cosmetic surgery, or other changes that anindividual's body may undergo. Furthermore, the augmented reality device102 may display other virtual images 114 that show how musculoskeletalfeatures would function under the certain changes.

With reference now to FIG. 2, in some implementations, an exemplarysystem 200 for displaying augmented anatomical features (e.g., virtualimages 114 a-114 c) provides the user 104 (e.g., a healthcare provider)with access to an augmented reality module 204 to enhance a user's 104view of a target individual 106. The system 200 may include a network208 that provides access to the augmented reality module 204 thatprovides for the performance of services on remote devices. Accordingly,the network 208 allows for interaction between the user 104 and theaugmented reality module 204. For instance, the augmented reality module204 may provide the user 104 access to the augmented reality module 204and receive event data inputted by the user 104 associated with theuser's 104 interaction with the augmented reality module 204. In turn,augmented reality module 204 may store event data in a storage resource.

The network 208 may include any type of network that allows sending andreceiving communication signals, such as a wireless telecommunicationnetwork, a cellular telephone network, a time division multiple access(TDMA) network, a code division multiple access (CDMA) network, Globalsystem for mobile communications (GSM), a third generation (3G) network,fourth generation (4G) network, a satellite communications network, andother communication networks. The network 208 may include one or more ofa Wide Area Network (WAN), a Local Area Network (LAN), and a PersonalArea Network (PAN). In some examples, the network 208 includes acombination of data networks, telecommunication networks, or acombination of data and telecommunication networks. An augmented realitydevice 102 and augmented reality module 204 communicate with each otherby sending and receiving signals (wired or wireless) via the network208. In some examples, the network 208 provides access to cloudcomputing resources, which may be elastic/on-demand computing and/orstorage resources available over the network 208. The term ‘cloud’services generally refers to a service performed not locally on a user'sdevice (e.g., device 102), but rather delivered from one or more remotedevices accessible via one or more networks 208.

The augmented reality device 102 may include, but is not limited to, aportable electronic device (e.g., smartphone, cellular phone, personaldigital assistant, personal computer, or wireless tablet device), awearable augmented reality device, or any other electronic devicecapable of capturing images and overlaying computer-generated or virtualimages (e.g., virtual images 114 a-114 c) over a real world view (e.g.,the visual representation 105). The augmented reality device 102includes data processing hardware (a computing device that executesinstructions), memory hardware, and a display 110 in communication withthe data processing hardware. Input module 214 provides the user 104access to interacting with the augmented reality module 204 through theaugmented reality device 102. In some examples, the input module 214includes a keyboard, touchpad, mouse, microphones, eye-tracking device,gesture tracking device, and/or a camera for allowing the user 104 toinput data. In addition to or in lieu of the display 110, augmentedreality device 102 may include one or more speakers to output audio datato the user 104.

In some implementations, the user 104 may interact with the input module214 by inputting data corresponding to reference markers 224. Thereference markers 224 may correspond to locations on the targetindividual 106. Data corresponding to the reference markers 224 is thensent to the augmented reality module 204. The augmented reality module204 may communicate with an anatomy module 216. For instance, theaugmented reality module 204 may send anatomical data 222 correspondingto the reference markers 224 to the anatomy module 216. The augmentedreality module 204 may then request data corresponding to visualrepresentations (e.g., virtual images 114 a-114 c) of anatomicalfeatures from the anatomy module 216. The anatomy module 216 may thenretrieve data corresponding to the visual representations of anatomicalfeatures from the anatomical database 218. The anatomy module 216 maythen generate an anatomical profile 220 to be displayed on the display110.

The augmented reality device 102 may include the image capture device108 having a detection module 210 and a tracking module 212. Thedetection module 210 and tracking module 212 may obtain visual datacorresponding to the target individual 106 and send it to the augmentedreality module 204. The visual data may be data corresponding to thecurrent real world view of the image capture device 108 and may includedata such as the distance between the target individual 106 and theaugmented reality device 102, whether the target individual 106 is inmotion or at rest, or any other data that corresponds to thevisualization of the target individual 106 by the augmented realitydevice 102. The image capture device 108 may send data from thedetection module 210 and the tracking module 212 to the augmentedreality module 204. The augmented reality module 204 may use the datafrom the detection module 210 and the tracking module 212 correspondingto the image capture device's 108 real world view, coupled with virtualimages of the anatomical profile 216 to create a composite enhanced viewof the target individual 106 on the display 110.

FIG. 3 is a flow chart illustrating a method 300 for displayingaugmented anatomical features (e.g., virtual images 114 a-114 c) inaccordance with an example implementation of the disclosed technology.According to one example, the method 300 may be performed by anelectronic device, such as the augmented reality device 102. The method300 begins at block 302 where the augmented reality device detects atarget individual (e.g., target individual 106) and tracks the targetindividual. The target individual has a body. For example, at block 302,the image capture device 108 a, 108 b may detect and/or track the targetindividual 106 via the detection module 210 and/or the tracking module212.

At block 304, the method includes displaying, on a display (e.g.,display 110), a visual representation of the body. For example, at block302, the display 110 a, 110 b may receive and display the visualrepresentation 105, 105′ from the augmented reality module 204. In someimplementations, after block 304, the method may return to block 302. Inother implementations, the method may advance to block 306.

At block 306, the method includes identifying a plurality of referencemarkers (e.g., reference markers 112) on the visual representation(e.g., the visual representation 105, 105′) of the body. For example,the augmented reality device 102 may receive the reference markers 112from the user's interaction with the display 110 a, 110 b, machinelearning, or another previously-described method. In someimplementations, the augmented reality device 102 may receive an inputof reference markers 112 from the user 104 and receive and generate thegraphical representation 113 from the augmented reality module 204.

At block 308, the method includes determining, at a processor, ananatomical profile (e.g., anatomical profile 216) of the targetindividual based on the plurality of reference markers. The anatomicalprofile may include a plurality of characteristics (e.g., age, gender,etc.) corresponding to the target individual. In some implementations,the augmented reality device 102 receives an anatomical profile based onthe reference markers 112. For example, the anatomy module 216 maytransmit the anatomical profile to the augmented reality device 102through the augmented reality module 204.

At block 310, the method includes displaying, on the display, graphicalrepresentations (e.g., virtual images 114 a-114 c) of the anatomicalfeatures overlaid on the visual representation 105, 105′ of the body.The graphical representations of the anatomical features may be orientedon the visual representation of the body based on the anatomicalprofile. In some implementations, the augmented reality device generatesand displays a graphical representation of the anatomical profileoverlaid on the visual representation of the target individual.Following block 310, the method 300 may conclude.

FIG. 4 is schematic view of an example computing device 400 that may beused to implement the systems and methods described in this document.The computing device 400 is intended to represent various forms ofdigital computers, such as laptops, desktops, workstations, personaldigital assistants, servers, blade servers, mainframes, and otherappropriate computers. The components shown here, their connections andrelationships, and their functions, are meant to be exemplary only, andare not meant to limit implementations of the inventions describedand/or claimed in this document.

The computing device 400 includes a processor 410, memory 420, a storagedevice 430, a high-speed interface/controller 440 connecting to thememory 420 and high-speed expansion ports 450, and a low speedinterface/controller 460 connecting to a low speed bus 470 and a storagedevice 430. Each of the components 410, 420, 430, 440, 450, and 460, areinterconnected using various busses, and may be mounted on a commonmotherboard or in other manners as appropriate. The processor 410 canprocess instructions for execution within the computing device 400,including instructions stored in the memory 420 or on the storage device430 to display graphical information for a graphical user interface(GUI) on an external input/output device, such as display 480 coupled tohigh speed interface 440. In other implementations, multiple processorsand/or multiple buses may be used, as appropriate, along with multiplememories and types of memory. Also, multiple computing devices 400 maybe connected, with each device providing portions of the necessaryoperations (e.g., as a server bank, a group of blade servers, or amulti-processor system).

The memory 420 stores information non-transitorily within the computingdevice 400. The memory 420 may be a computer-readable medium, a volatilememory unit(s), or non-volatile memory unit(s). The non-transitorymemory 420 may be physical devices used to store programs (e.g.,sequences of instructions) or data (e.g., program state information) ona temporary or permanent basis for use by the computing device 400.Examples of non-volatile memory include, but are not limited to, flashmemory and read-only memory (ROM)/programmable read-only memory(PROM)/erasable programmable read-only memory (EPROM)/electronicallyerasable programmable read-only memory (EEPROM) (e.g., typically usedfor firmware, such as boot programs). Examples of volatile memoryinclude, but are not limited to, random access memory (RAM), dynamicrandom access memory (DRAM), static random access memory (SRAM), phasechange memory (PCM) as well as disks or tapes.

The storage device 430 is capable of providing mass storage for thecomputing device 400. In some implementations, the storage device 430 isa computer-readable medium. In various different implementations, thestorage device 430 may be a floppy disk device, a hard disk device, anoptical disk device, or a tape device, a flash memory or other similarsolid state memory device, or an array of devices, including devices ina storage area network or other configurations. In additionalimplementations, a computer program product is tangibly embodied in aninformation carrier. The computer program product contains instructionsthat, when executed, perform one or more methods, such as thosedescribed above. The information carrier is a computer- ormachine-readable medium, such as the memory 420, the storage device 430,or memory on processor 410.

The high speed controller 440 manages bandwidth-intensive operations forthe computing device 400, while the low speed controller 460 manageslower bandwidth-intensive operations. Such allocation of duties isexemplary only. In some implementations, the high-speed controller 440is coupled to the memory 420, the display 480 (e.g., through a graphicsprocessor or accelerator), and to the high-speed expansion ports 450,which may accept various expansion cards (not shown). In someimplementations, the low-speed controller 460 is coupled to the storagedevice 430 and a low-speed expansion port 490. The low-speed expansionport 490, which may include various communication ports (e.g., USB,Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or moreinput/output devices, such as a keyboard, a pointing device, a scanner,or a networking device such as a switch or router, e.g., through anetwork adapter.

The computing device 400 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 400 a or multiple times in a group of such servers 400a, as a laptop computer 400 b, or as part of a rack server system 400 c.

Among other advantages, the present disclosure provides methods, userdevices, and systems for displaying augmented anatomical features. Anaugmented reality device may overlay virtual images of anatomy on top ofthe human body illustrate an approximation of the structures, tissues ororgans that lie beneath the surface of an individual, such as a targetindividual, in front of a user, such as a healthcare professional. Thevirtual images can be adjusted to fit the target individual. The usermay use the augmented reality device to identify certain anatomicalreference points on the body of the target individual, and use thosepoints to anchor and adjust the virtual images over the targetindividual. The virtual images may be representative of human anatomy ofa human of similar age, sex, etc.

Among other advantages, the present disclosure also provides a method,user device, and system that does not require input of data files fromoutside imaging (e.g., x-ray, magnetic resonance imaging, computedtomography scan, etc.). Such files may have incompatible formats, belarge and unwieldy, or require a large amount of processing power foreach target individual.

Among other advantages, the present disclosure also provides a method,user device, and system that may be for general use. In this regard, useof the augmented reality device may not be restricted to certifiedhealthcare providers. Furthermore, the expectation of the augmentedreality device may be to output or display a computer-generatedapproximation of a representative human anatomy.

Among other advantages, the present disclosure also provides broadapplicability. The augmented reality device may be in constant and rapiduse with one target individual after another, and without requiring theinput of outside data.

Various implementations of the systems and techniques described hereincan be realized in digital electronic and/or optical circuitry,integrated circuitry, specially designed ASICs (application specificintegrated circuits), computer hardware, firmware, software, and/orcombinations thereof. These various implementations can includeimplementation in one or more computer programs that are executableand/or interpretable on a programmable system including at least oneprogrammable processor, which may be special or general purpose, coupledto receive data and instructions from, and to transmit data andinstructions to, a storage system, at least one input device, and atleast one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium” and“computer-readable medium” refer to any computer program product,non-transitory computer readable medium, apparatus and/or device (e.g.,magnetic discs, optical disks, memory, Programmable Logic Devices(PLDs)) used to provide machine instructions and/or data to aprogrammable processor, including a machine-readable medium thatreceives machine instructions as a machine-readable signal. The term“machine-readable signal” refers to any signal used to provide machineinstructions and/or data to a programmable processor.

The processes and logic flows described in this specification can beperformed by one or more programmable processors, also referred to asdata processing hardware, executing one or more computer programs toperform functions by operating on input data and generating output. Theprocesses and logic flows can also be performed by special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit). Processors suitable for theexecution of a computer program include, by way of example, both generaland special purpose microprocessors, and any one or more processors ofany kind of digital computer. Generally, a processor will receiveinstructions and data from a read only memory or a random access memoryor both. The essential elements of a computer are a processor forperforming instructions and one or more memory devices for storinginstructions and data. Generally, a computer will also include, or beoperatively coupled to receive data from or transfer data to, or both,one or more mass storage devices for storing data, e.g., magnetic,magneto optical disks, or optical disks. However, a computer need nothave such devices. Computer readable media suitable for storing computerprogram instructions and data include all forms of non-volatile memory,media and memory devices, including by way of example semiconductormemory devices, e.g., EPROM, EEPROM, and flash memory devices; magneticdisks, e.g., internal hard disks or removable disks; magneto opticaldisks; and CD ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, one or more aspects of thedisclosure can be implemented on a computer having a display device,e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, ortouch screen for displaying information to the user and optionally akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

A software application (i.e., a software resource) may refer to computersoftware that causes a computing device to perform a task. In someexamples, a software application may be referred to as an “application,”an “app,” or a “program.” Example applications include, but are notlimited to, system diagnostic applications, system managementapplications, system maintenance applications, word processingapplications, spreadsheet applications, messaging applications, mediastreaming applications, social networking applications, and gamingapplications.

The non-transitory memory may be physical devices used to store programs(e.g., sequences of instructions) or data (e.g., program stateinformation) on a temporary or permanent basis for use by a computingdevice. The non-transitory memory may be volatile and/or non-volatileaddressable semiconductor memory. Examples of non-volatile memoryinclude, but are not limited to, flash memory and read-only memory(ROM)/programmable read-only memory (PROM)/erasable programmableread-only memory (EPROM)/electronically erasable programmable read-onlymemory (EEPROM) (e.g., typically used for firmware, such as bootprograms). Examples of volatile memory include, but are not limited to,random access memory (RAM), dynamic random access memory (DRAM), staticrandom access memory (SRAM), phase change memory (PCM) as well as disksor tapes.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. Accordingly, otherimplementations are within the scope of the following claims.

What is claimed is:
 1. A method comprising: detecting a targetindividual having a body; tracking the target individual; displaying, ona display, a visual representation of the body; identifying a pluralityof reference markers on the visual representation of the body based ongestures within an environment of the target individual, each gesturecorresponding to one or more particular parts of the body of the targetindividual and being captured by an image capture device incommunication with the display; determining, at a processor, ananatomical profile of the target individual based on the plurality ofreference markers, the anatomical profile including a plurality ofanatomical features; and displaying, on the display, graphicalrepresentations of the anatomical features overlaid on the visualrepresentation of the body, the graphical representations of theanatomical features being oriented on the visual representation of thebody based on the anatomical profile.
 2. The method of claim 1, whereinthe plurality of reference markers correspond to at least one of anavel, a portion of a sternum, a portion of a hip, a portion of acollarbone, or a portion of a shoulder.
 3. The method of claim 1,wherein the anatomical profile of the target individual is further basedon a plurality of data corresponding to the body.
 4. The method of claim1, wherein the plurality of anatomical features includes at least one oforgans, bones, muscles, or blood vessels.
 5. The method of claim 1,wherein the graphical representations correspond to a future state ofthe target individual, the future state comprising changes to theanatomical features.
 6. A user device comprising: a display; an imagecapture device in communication with the display; data processinghardware in communication with the display; and memory hardware incommunication with the data processing hardware, the memory hardwarestoring instructions that when executed on the data processing hardwarecause the data processing hardware to perform operations including:detecting a target individual having a body; tracking the targetindividual; displaying, on the display, a visual representation of thebody; identifying a plurality of reference markers on the visualrepresentation of the body based on gestures within an environment ofthe target individual, each gesture corresponding to one or moreparticular parts of the body of the target individual and being capturedby the image capture device in communication with the display;determining an anatomical profile of the target individual based on theplurality of reference markers, the anatomical profile including aplurality of anatomical features; and displaying, on the display,graphical representations of the anatomical features overlaid on thevisual representation of the body, the graphical representations of theanatomical features being oriented on the visual representation of thebody based on the anatomical profile.
 7. The user device of claim 6,wherein the plurality of reference markers correspond to at least one ofa navel, a portion of a sternum, a portion of a hip, a portion of acollarbone, or a portion of a shoulder.
 8. The user device of claim 6,wherein the anatomical profile of the target individual is further basedon a plurality of data corresponding to the body.
 9. The user device ofclaim 6, wherein the plurality of anatomical features includes at leastone of organs, bones, muscles, or blood vessels.
 10. The user device ofclaim 6, wherein the plurality of reference markers are identified bythe data processing hardware.
 11. The user device of claim 6, whereinthe anatomical profile of the target individual includes a defaultanatomical profile that is modified based on the plurality of referencemarkers.
 12. The user device of claim 6, wherein the target individualis disposed within the environment.
 13. The user device of claim 6,wherein identifying the plurality of reference markers on the visualrepresentation of the body based on gestures within the environment ofthe target individual includes identifying the plurality of referencemarkers on the visual representation of the body based on hand gestureswithin the environment of the target individual.
 14. A systemcomprising: a user device including: a first display; data processinghardware in communication with the first display; memory hardware incommunication with the data processing hardware, the memory hardwarestoring instructions that when executed on the data processing hardwarecause the data processing hardware to perform operations including:detecting a target individual having a body; tracking the targetindividual; displaying, on the first display, a visual representation ofthe body; identifying a plurality of reference markers on the visualrepresentation of the body based on gestures within an environment ofthe target individual, each gesture corresponding to one or moreparticular parts of the body of the target individual and being capturedby an image capture device in communication with the first display; anddetermining an anatomical profile of the target individual based on theplurality of reference markers, the anatomical profile including aplurality of anatomical features; and a wearable in communication withthe user device, the wearable including a second display configured todisplay graphical representations of the anatomical features overlaid onthe visual representation of the body, the graphical representations ofthe anatomical features being oriented on the visual representation ofthe body based on the anatomical profile.
 15. The system of claim 14,wherein the plurality of reference markers correspond to at least one ofa navel, a portion of a sternum, a portion of a hip, a portion of acollarbone, or a portion of a shoulder.
 16. The system of claim 14,wherein the anatomical profile of the target individual is further basedon a plurality of data corresponding to the body.
 17. The system ofclaim 14, wherein the plurality of anatomical features includes at leastone of organs, bones, muscles, or blood vessels.
 18. The system of claim14, wherein the wearable is further configured to detect and track thetarget individual independently of the user device.
 19. The system ofclaim 14, wherein the graphical representations correspond to a futurestate of the target individual, the future state comprising changes tothe anatomical features.